ixl FreeBSD* Base Driver and ixlv VF Driver for the
Intel XL710 Ethernet Controller Family
================================================================
August 26, 2014

Contents
========

- Overview
- Supported Adapters
- The VF Driver
- Building and Installation
- Additional Configurations
- Known Limitations


Overview
--------
This file describes the FreeBSD* driver for Intel(R) Ethernet. This driver has
been developed for use with all community-supported versions of FreeBSD.

For questions related to hardware requirements, refer to the documentation
supplied with your Intel Ethernet Adapter. All hardware requirements listed
apply to use with FreeBSD.


Identifying Your Adapter
------------------------
The driver in this release is compatible with XL710 and X710-based Intel
Ethernet Network Connections.

For information on how to identify your adapter, go to the Adapter &
Driver ID Guide at:
http://support.intel.com/support/go/network/adapter/proidguide.htm

For the best performance, make sure the latest NVM/FW is installed on
your device and that you are using the newest drivers.

For the latest NVM/FW images and Intel network drivers, refer to the 
following website and select your adapter.
http://www.intel.com/support


SFP+ Devices with Pluggable Optics
----------------------------------

SR Modules
----------
IntelDUAL RATE 1G/10G SFP+ SR (bailed)E10GSFPSR

LR Modules
---------- 
IntelDUAL RATE 1G/10G SFP+ LR (bailed)E10GSFPLR

1G SFPModules
--------------
The following is a list of 3rd party SFP modules that have received some
testing. Not all modules are applicable to all devices.

Supplier  Type Part Numbers
Finisar  1000BASE-T SFP FCLF-8251-3
Kinnex A 1000BASE-T SFP XSFP-T-RJ12-0101-DLL
Avago 1000BASE-T SFP ABCU-5710RZ

QSFP+ Modules
-------------
IntelTRIPLE RATE 1G/10G/40G QSFP+ SR (bailed)E40GQSFPSR
 QSFP+ 1G speed is not supported on XL710 based devices.

X710/XL710 Based SFP+ adapters support passive QSFP+ Direct Attach cables.
Intel recommends using Intel optics and cables. Other modules may function
but are not validated by Intel. Contact Intel for supported media types.


The VF Driver
=============
The VF driver is normally used in a virtualized environment where a host driver
manages SRIOV, and provides a VF device to the guest. With this release, the
only host environment tested was using Linux QEMU/KVM. Support is planned for
Xen and VMWare hosts at a later time.

 In the FreeBSD guest the IXLV driver would be loaded and will function using
the VF device assigned to it.

 The VF driver provides most of the same functionality as the CORE driver, but
is actually a slave to the Host, access to many controls is accomplished by a
request to the Host via what is called the "Admin queue". These are startup and
initialization events, however, once in operation the device is self-contained
and should achieve near native performance. 

Some notable limitations of the VF environment: for security reasons the driver
is never permitted to be promiscuous, therefore a tcpdump will not behave the
same with the interface. Second, media info is not available from the PF, so it
will always appear as auto.
Building and Installation 
-------------------------

 NOTE: This driver package is to be used only as a standalone archive and the
user should not attempt to incorporate it into the kernel source tree.
In the instructions below, x.x.x is the driver version as indicated in the name
of the driver tar file.

 Move the base driver tar file to the directory of your choice. For 
 example, use /home/username/ixl or /usr/local/src/ixl. 

Untar/unzip the archive:

tar xzf ixl-x.x.x.tar.gz 

This will create the ixl-x.x.x directory.

To install man page:

cd ixl-x.x.x
gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz

To load the driver onto a running system:

cd ixl-x.x.x/src
make load

To assign an IP address to the interface, enter the following:

ifconfig ixl<interface_num> <IP_address>

Verify that the interface works. Enter the following, where <IP_address>
is the IP address for another machine on the same subnet as the interface
that is being tested:

ping <IP_address>

If you want the driver to load automatically when the system is booted:

cd ixl-x.x.x/src
make
make install

Edit /boot/loader.conf, and add the following line:
if_ixl_load="YES"

Edit /etc/rc.conf, and create the appropriate ifconfig_ixl<interface_num> entry:

ifconfig_ixl<interface_num>="<ifconfig_settings>"

Example usage:
ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0"

NOTE: For assistance, see the ifconfig man page.


Configuration and Tuning
========================

 Important system configuration changes:
---------------------------------------

-Change the file /etc/sysctl.conf, and add the line: 

hw.intr_storm_threshold: 0 (the default is 1000)

-Best throughput results are seen with a large MTU; use 9706if possible.
-The default number of descriptors per ring is 1024, increasing this may
 improve performance, depending on your use case.
- If you have a choice, run on a 64-bit OS rather than a 32-bit OS.

Jumbo Frames
------------
Jumbo Frames support is enabled by changing the Maximum Transmission Unit
(MTU) to a value larger than the default value of 1500.

Use the ifconfig command to increase the MTU size. For example, enter the
following where <x> is the interface number:

ifconfig eth<x> mtu 9000

To confirm an interface's MTU value, use the ifconfig command.

To confirm the MTU used between two specific devices, use:

route get <destination_IP_address>

  
NOTES:
- The maximum MTU setting for Jumbo Frames is 9706. This value coincides
with the maximum Jumbo Frames size of 9728 bytes.
- This driver will attempt to use multiple page sized buffers to receive 
each jumbo packet. This should help to avoid buffer starvation issues 
when allocating receive packets.


VLANS
-----
To create a new VLAN interface:

ifconfig <vlan_name> create

To associate the VLAN interface with a physical interface and
assign a VLAN ID, IP address, and netmask:

ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan
<vlan_id> vlandev <physical_interface>

Example:

ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0

In this example, all packets will be marked on egress with 802.1Q VLAN tags,
specifying a VLAN ID of 10.

To remove a VLAN interface:
ifconfig <vlan_name> destroy


Checksum Offload
----------------

Checksum offloading supports both TCP and UDP packets and is supported for both
transmit and receive.

Checksum offloading can be enabled or disabled using ifconfig. Both transmit and
receive offloading will be either enabled or disabled together. You cannot
enable/disable one without the other.

To enable checksum offloading:
ifconfig ixlX rxcsum

To disable checksum offloading:
ifconfig ixlX -rxcsum

To confirm the current setting:
ifconfig ixlX

Look for the presence or absence of the following line:
options=3 <RXCSUM,TXCSUM>

See the ifconfig man page for further information.


TSO
---
TSO (TCP Segmentation Offload) supports both IPv4 and IPv6. TSO can be 
disabled and enabled using the ifconfig utility or sysctl.

NOTE: TSO requires Tx checksum, if Tx checksum is disabled, TSO will also 
be disabled.

To enable/disable TSO in the stack:
sysctl net.inet.tcp.tso=0 (or 1 to enable it)

Doing this disables/enables TSO in the stack and affects all installed adapters.

To disable BOTH TSO IPv4 and IPv6:
ifconfig ixl<interface_num> -tso

To enable BOTH TSO IPv4 and IPv6:
ifconfig ixl<interface_num> tso

You can also enable/disable IPv4 TSO or IPv6 TSO individually. Simply replace
tso|-tso in the above command with tso4 or tso6. For example, to disable
TSO IPv4:
ifconfig ixl<interface_num> -tso4


LRO
---

LRO (Large Receive Offload) may provide rx performance improvement.
However, it is incompatible with packet-forwarding workloads. You should
carefully evaluate the environment and enable LRO when possible.

To enable:

ifconfig ixl<interface_num> lro

It can be disabled by using:

ifconfig ixl<interface_num> -lro


Flow Control
------------

Ethernet Flow Control (IEEE 802.3x) can be configured with sysctl to enable
receiving and transmitting pause frames for ixl. When transmit is enabled,
pause frames are generated when the receive packet buffer crosses a predefined
threshold. When receive is enabled, the transmit unit will halt for the time
delay specified when a pause frame is received. 

Flow Control is disabled by default.

Use sysctl to change the flow control settings.

To enable flow control to Rx pause frames:
sysctl dev.ixl.<interface_num>.fc=1

To enable flow control to Tx pause frames:
sysctl dev.ixl.<interface_num>.fc=2

To enable flow control to Rx and Tx pause frames:
sysctl dev.ixl.<interface_num>.fc=3

To disable flow control:
sysctl dev.ixl.<interface_num>.fc=0

NOTE: You must have a flow control capable link partner.

NOTE: The VF driver does not have access to flow control, it must be
managed from the host side.


Known Issues/Troubleshooting
----------------------------


Network Memory Buffer allocation
--------------------------------
FreeBSD may have a low number of network memory buffers (mbufs) by default.
If your mbuf value is too low, it may cause the driver to fail to initialize
and/or cause the system to become unresponsive. You can check to see if the
system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs
by editing the lines below in /etc/sysctl.conf:

kern.ipc.nmbclusters
kern.ipc.nmbjumbop
kern.ipc.nmbjumbo9
kern.ipc.nmbjumbo16
kern.ipc.nmbufs

The amount of memory that you allocate is system specific, and may require
some trial and error. Also, increasing the following in /etc/sysctl.conf
could help increase network performance:

kern.ipc.maxsockbuf
net.inet.tcp.sendspace
net.inet.tcp.recvspace
net.inet.udp.maxdgram
net.inet.udp.recvspace

UDP Stress Test Dropped Packet Issue
------------------------------------

Under small packet UDP stress with the ixldriver, the system may
drop UDP packets due to socket buffers being full. Setting the driver Flow
Control variables to the minimum may resolve the issue. 


Disable LRO when routing/bridging
---------------------------------

LRO must be turned off when forwarding traffic.


Lower than expected performance
-------------------------------

Some PCIe x8 slots are actually configured as x4 slots. These slots have
insufficient bandwidth for full line rate with dual port and quad port
devices. In addition, if you put a PCIe Generation 3-capable adapter
into a PCIe Generation 2 slot, you cannot get full bandwidth. The driver
detects this situation and writes the following message in the system log:

"PCI-Express bandwidth available for this card is not sufficient for optimal
performance. For optimal performance a x8 PCI-Express slot is required."

If this error occurs, moving your adapter to a true PCIe Generation 3 x8 slot
 will resolve the issue.


Support
-------
For general information, go to the Intel support website at:
www.intel.com/support/

or the Intel Wired Networking project hosted by Sourceforge at:
http://sourceforge.net/projects/ixl
If an issue is identified with the released source code on a supported
kernel with a supported adapter, email the specific information related to the
issue to freebsdnic@mailbox.intel.com


License
-------

This program is free software; you can redistribute it and/or modify it under
the terms and conditions of the GNU General Public License, version 2, as
published by the Free Software Foundation.

This program is distributed in the hope it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
St - Fifth Floor, Boston, MA 02110-1301 USA.

The full GNU General Public License is included in this distribution in the
file called "COPYING".

Intel(R) XL710/X710 Network Driver
Intel(R) XL710/X710 Virtual Function Network Driver
Copyright(c) 2014-2015 Intel Corporation.
